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1.  Scientific  Progress  Overview 

The  project  on  Tactile  Sensing  and  Information  Processing  for  Man  and  Machine  Systems  is 
completing  its  fifth  year.  Accomplishments  include  the  continued  development  of  tactile  sensors 
and  algorithms  for  sensor-based  exploration  and  control,  and  continued  development  of  models 
of  human  mechanoreception  in  manipulation. 

In  the  area  of  sensor  development,  researchers  at  Stanford  have  investigated  stylus  sensors  for 
surface  exploration.  Tests  were  conducted  with  NASA  Ames  (see  Technology  Transitions)  to 
evaluate  the  stylus  as  a  tool  for  remote  geological  exploration.  Very  high  resolution  tactile  arrays 
have  also  been  fabricated  and  tested.  The  arrays  are  CMOS-compatible  and  contain  circuitry  for 
scanning  or  individually  accessing  the  elements.  Sensitivities  are  approximately  5 1  mV/kPa  for 
normal  stresses  and  12  mV/kPa  for  shear  stresses.  An  improved  optical  sensor  has  been 
developed  for  measuring  local  object  geometries  been  developed  and  applied  to  the  design  of  a 
medical  probe. 

In  the  area  of  control  and  sensor  based  exploration,  robot  fingers  have  been  used  to  manipulate 
objects  of  unknown  shape  while  simultaneously  collecting  information  about  the  objects’ 
geometric  features 

In  the  area  of  human/machine  interaction,  experiments  were  conducted  to  determine  the  ability  of 
subjects  to  distinguish  fine  features  on  real  and  virtual  surfaces  and  to  accurately  detect  contacts 
with  virtual  objects  using  ungrounded  haptic  feedback. 

2.  Significant  Accomplishments 
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Comparison  of  Grounded  and 
Ungrounded  Haptic  Interfaces 


a)  Description 

Haptic  interfaces,  that  apply  forces  to  the  fingertips  of  a  human  operator,  can  be  classified  as 
grounded  or  ungrounded.  Grounded  devices,  such  as  SensAble  Devices’  Phantom  or 
Immersion’s  Impulse  Engine,  are  attached  to  a  stationary  object  such  as  a  desk.  When  the 
operator  touches  a  virtual  wall,  a  contact  force  is  applied  through  the  interface,  inhibiting  further 
motion.  For  ungrounded  devices  a  contact  force  is  felt  but  there  is  no  impediment  to  motion  of 
the  arm.  Little  prior  research  has  been  done  on  the  perceptual  effects  of  displaying  contacts  with 
virtual  objects  using  an  ungrounded  haptic  interface.  Experiments  were  conducted  at  Stanford  to 
compare  how  accurately  subjects  could  identify  contacts  with  virtual  walls  using  ungrounded 
versus  grounded  feedback.  Two  haptic  interfaces  were  constructed  and  operated  in  three  modes: 
with  grounded  force  feedback  applied  to  the  wrist,  with  ungrounded  forces  applied  to  the 
fingertips,  and  with  grounded  wrist  forces  in  addition  to  fingertip  forces.  Tests  were  conducted  to 
see  how  quickly  subjects  could  arrest  motion  upon  sensing  contact  (measured  as  virtual  boundary 
penetration)  and  how  accurately  they  could  distinguish  among  objects  of  different  size  (see  figure 
1)  [Richard  and  Cutkosky  1997]. 


Although  the  grounded  feedback  unsurprisingly  resulted  in  smaller  boundary  penetrations,  the 
results  and  subjects’  comments  showed  that  ungrounded  force  feedback  was  effective  at 
rendering  virtual  object  surfaces.  With  ungrounded  feedback  subjects  had  no  difficulty  in 
distinguishing  among  objects  with  a  difference  in  size  of  0.5  cm  and  greater.  The  size 
discrimination  experiments  were  subsequently  repeated  using  a  prototype  device  from  Virtual 
Technologies  Corp.  (see  figure  2),  used  to  teleoperate  a  two-fingered  robot  hand.  Despite  the 
compliance  present  in  the  ungrounded  Cybergrasp  prototype,  subjects  were  again  able  to 
distinguish  among  objects  whose  dimensions  differed  by  0.5  cm  or  greater. 

b)  Significance 

Ungrounded  haptic  interfaces,  which  can  be  worn  on  the  arms  and  wrists,  are  attractive  for 
virtual  reality  applications  because  they  can  be  made  minimally  encumbering  as  compared  to 
stationary,  grounded  devices.  However,  the  ability  of  ungrounded  devices  to  effectively  display 
contact  forces  with  virtual  objects  has  been  a  subject  of  concern.  The  experiments  conducted  at 
Stanford  reveal  that  while  grounded  force  feedback  is  unsurprisingly  better  at  reducing  artificial 
effects  such  as  the  penetration  of  virtual  object  surfaces,  ungrounded  force  feedback  is 
nonetheless  effective  for  displaying  contacts.  The  effectiveness  of  ungrounded  feedback  is 
improved  when  visual  feedback,  and  cutaneous  feedback  consisting  of  contact  forces  applied  at 
the  fingertips,  are  part  of  the  package.  However,  it  is  essential  that  time  delays  associated  with 
these  different  channels  be  equalized. 

c)  Figures 

Figure  1.  A  comparison  of  grounded  versus  ungrounded  haptic  feedback.  How  effective  is 
ungrounded  feedback  for  displaying  contacts  with  virtual  objects? 

Figure  2.  Example  of  ungrounded  haptic  feedback  being  used  for  teleoperation  of  a  two-fingered 
robot  hand.  The  device  is  a  prototype  of  Cybergrasp,  developed  at  Virtual  Technologies  Inc.  by 
Dr.  Marc  Tremblay,  an  alumnus  of  the  ONR  project  on  Tactile  Sensing  and  Information 
Processing  for  Man  and  Machine  Systems. 

Figure  3.  Boundary  discrimination  tests  with  different  modes  of  haptic  feedback:  (A) 
ungrounded  fingertip  forces,  (B)  grounded  feedback  applied  to  the  wrist,  (C)  grounded  wrist 
feedback  +  fingertip  forces.  Although  grounded  feedback  worked  best  for  objects  of  very  similar 
size,  subjects  had  little  difficulty  with  differences  of  0.5  cm  or  greater  using  ungrounded 
feedback  [Richard  and  Cutkosky  1997]. 
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Image-Based  Tactile  Sensing 

a)  Description 

During  the  past  year,  we  have  developed  and  implemented  the  design  of  a  novel  image-based 
tactile  sensor.  Besides  being  able  to  localize  contact  position,  this  sensor  can  recover 
information  about  contact  shape  as  well.  Unlike  the  majority  of  existing  haptic  technologies,  this 
sensor  uses  vision-based  methodologies.  The  original  design  for  the  sensor  (see  figure  4(a)  in  the 
Appendix)  used  a  ccd  camera,  pinhole,  and  led  lighting.  The  current  design  (see  figure  4(b)  in 
the  Appendix)  uses  a  more  robust  microcamera  with  lens  and  fiber  optic  lighting. 

The  operation  of  the  sensor  is  based  on  using  the  camera  to  image  the  inside  surface  of  the  sensor 
membrane  tip.  The  membrane  is  filled  with  clear  liquid-like  silicon  RTV  and  sealed  from  the 
electronics  by  a  clear  window.  A  pattern  of  dots  is  drawn  on  the  inner  surface  of  the  membrane. 
As  an  object  makes  contact  and  deforms  the  sensor  membrane,  the  pattern  of  dots  deforms  as 
well.  Using  the  mechanics  of  the  membrane  as  well  as  constant  volume  and  minimum  energy 
requirements  for  the  deformed  membrane  surface,  our  algorithm  computes  the  shape  of  the 
tactile  surface  as  it  is  being  deformed.  A  typical  example  of  a  pattern  seen  by  the  sensor  camera 
together  with  the  reconstructed  3-D  membrane  surface  are  given  in  figure  5  (in  the  Appendix.) 
Our  current  software  implementation  allows  us  to  reconstruct  the  sensor  surface  at  a  rate  of  7Hz, 
a  significant  increase  from  the  speed  of  IHz  obtained  a  year  ago. 

b)  Signiflcance 

Dexterous  robots  have  been  widely  studied  and  developed  over  the  past  decade  for  use  in 
dexterous  manipulation  of  objects  in  automated,  remote,  or  hostile  environments.  Tactile  sensing 
is  essential  for  such  robots,  much  as  it  is  for  humans  performing  dexterous  tasks.  In  the  past, 
tactile  surfaces  for  dexterous  robotic  hands  have  been  modeled  as  rigid  partly  due  to  a  lack  of 
deformable  tactile  sensors  and  partly  to  simplify  computations.  However,  manipulating  rigid 
objects  with  rigid  fingertips  is  neither  robust  nor  realistic.  Our  image-based  tactile  sensor 
behaves  much  like  a  human  fingertip  in  that  it  is  both  deformable  and  capable  of  localizing 
contact  in  three  dimensions.  Used  as  a  fingertip  at  the  end  of  robotic  fingers  (see  figure  3),  the 
sensor  provides  essential  haptic  information  and  vastly  improves  the  grasp  stability  and 
robustness. 

c)  Figures  (see  Appendix) 

3.  Productivity  Report 

(a)  papers  published  in  refereed  journals  . 

A.  Z.  Hajian  and  R.  D.  Howe,  “Identification  of  the  mechanical  impedance  at  the  human  finger 
ASME  Journal  of  Biomechanical  Engineering,  v  119  n  1  Feb  1997.  pp  109-114.. 
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BJ.  Kane,  M.R.  Cutkosky  and  G.T.A.  Kovacs,  “CMOS-compatible  traction  stress  sensor  for  use 
in  high-resolution  tactile  imaging,”  Sensors  and  Actuators,  v.  A  54,  n.  1-3,  June  1996,  pp.  511- 
516. 

R.D.  Howe  and  M.R.  Cutkosky,  “Practical  Force-Motion  Models  for  Sliding  Manipulation,”  The 
International  Journal  of  Robotics  Research,  v.  15,  n.  6,  December  1996,  pp.  557  -572. 

J.S.  Son,  M.R.  Cutkosky  and  R.D.  Howe,*  “Comparison  of  Contact  Sensor  Localization  Abilities 
During  Manipulation,”  Robotics  and  Autonomous  Systems,  v.  17,  n.  3,  June  1996,  pp.  217-233. 

I.  Kao,  M.R.  Cutkosky,  B.  Edin,  G.  Westling  and  R.  Johansson,  “Robotic  Stiffness  Control  and 
Calibration  as  Applied  to  Human  Grasping  Tasks,”  IEEE  Transactions  on  Robotics  and 
Automation,  v.  12,  n.  4,  August,  1997,  pp.557-566. 

(b)  papers  accepted  for  publication  in  refereed  journals 

A.  Z.  Hajian,  D.  S.  Sanchez,  and  R.  D.  Howe,  “Drum  roll;  Increasing  bandwidth  through  passive 
impedance  modulation,”  accepted  for  publication  in  the  IEEE  Transactions  on  Robotics  and 
Automation,  1997. 

W.  J.  Peine,  K.  C.  Foucher,  and  R.  D.  Howe,  “Finger  speed  during  single  digit  palpation,” 
accepted  for  publication  in  Human  Factors,  1997. 


D.  T.  V.  Pawluk,  J.  S.  Son,  P.  S.  Wellman,  W.  J.  Peine,  and  R.  D.  Howe.,  “A  Distributed 
Pressure  Sensor  For  Biomechanical  Measurements,”  ASME  Journal  of  Biomechanical 
Engineering,  in  press,  1997. 

(c)  technical  reports  and  conference  articles 

C.  Richard  and  M.R.  Cutkosky,  “Contact  Force  Perception  with  an  Ungrounded  Haptic 
Interface”,  1997  ASME  IMECE  6th  Annual  Symposium  on  Haptic  Interfaces,  Dallas,  TX,  Nov. 
15-21. 

A.  West  and  M.R.  Cutkosky,  “Detection  of  Real  and  Virtual  Fine  Surface  Features  with  a  Haptic 
Interface  and  Stylus,”  1997  ASME  IMECE  6th  Annual  Symposium  on  Haptic  Interfaces,  Dallas, 
TX,Nov.  15-21. 

C.  Richard,  A.M.  Okamura,  M.R.  Cutkosky,  “Getting  a  Feel  for  Dynamics;  using  haptic  interface 
kits  for  teaching  dynamics  and  controls,”  1997  ASME  IMECE  6th  Annual  Symposium  on  Haptic 
Interfaces,  Dallas,  TX,  Nov.  15-21. 

A.M.  Okamura,  M.L  Turner  and  M.R.  Cutkosky,  “Haptic  Exploration  of  Objects  with  Rolling 
and  Sliding,”  Proceedings  of  the  1997  IEEE  International  Conference  on  Robotics  and 
Automation,  Vol.  3,  April  20-25,  Albuquerque,  NM,  pp.  2485-2490. 


N.  J.  Ferrier,  K.  A.  Morgansen,  D.  Hristu.  “Implementation  of  Membrane  Shape 
Reconstruction.”  Tech.  Report  97-1,  Harvard  Robotics  Lab,  Harvard  University,  1997. 

W.J.  Peine,  P.S.  Wellman  and  R.  D.  Howe,  “Temporal  bandwidth  requirements  for  tactile  shape 
displays,”  to  be  presented  at  the  Sixth  Annual  Symposium  on  Haptic  Interfaces  for  Virtual 
Environment  and  Teleoperator  Systems,  ASME  International  Mechanical  Engineering  Congress 
and  Exposition,  Dallas,  Nov.  15-21,  1997. 

J.T.  Dennerlein,  P.  Millman,  and  R.D.  Howe,  “Vibrotactile  Feedback  for  Industrial 
Telemanipulators,”  to  be  presented  at  the  Sixth  Annual  Symposium  on  Haptic  Interfaces  for 
Virtual  Environment  and  Teleoperator  Systems,  ASME  International  Mechanical  Engineering 
Congress  and  Exposition,  Dallas,  Nov.  15-21,  1997. 


A.  Z.  Hajian,  D.  S.  Sanchez,  and  R.  D.  Howe,  “Drum  roll:  Increasing  bandwidth  through 
impedance  modulation,”  presented  at  the  IEEE  International  Conference  on  Robotics  and 
Automation,  Albuquerque,  New  Mexico,  April  20  -  25,  1997. 

P .  E.  Dupont,  T.  M.  Schulteis,  and  R.  D.  Howe  ,  “Experimental  Identification  of  Kinematic 
Constraints,”  presented  at  the  IEEE  International  Conference  on  Robotics  and  Automation, 
Albuquerque,  New  Mexico,  April  20  -  25, 1997. 

D.  T.V.  Pawluk  and  R.  D.  Howe,  “Contact  pressure  distribution  on  the  human  finger  pad,” 
presented  at  the  26th  Congress  of  the  International  Society  of  Biomechanics,  Tokyo,  August  25- 
29, 1997. 

D.  T.V.  Pawluk  and  R.  D.  Howe,  “Mechanical  impedance  and  energy  dissipation  in  the  human 
finger  pad,”  1997  ASME  Summer  Bioengineering  Conference,  Sun  River,  Oregon,  June  1997, 
BED-Vol.  35,  p.  591-592. 

P.S.  Wellman,  W.J.  Peine,  G.  Favalora,  and  R.D.  Howe,  “Mechanical  Design  and  Control  of  a 
High-Bandwidth  Shape  Memory  Alloy  Tactile  Display,”  Fifth  International  Symposium  on 
Experimental  Robotics,  Barcelona,  June  16-18, 1997,  Springer-Verlag  (in  press). 

(d)  books  or  book  chapters 

(e)  books  or  book  chapters  in  press 

(f)  patents  filed  or  granted 

(g)  invited  presentations 

D.  Hristu,  D.  A.  Kontarinis,  and  R.D.  Howe, 

6/30/96 

International  Federation  of  Automatic  Controls  World  Congress 
San  Francisco,  CA 

Title;  A  comparison  of  delay  and  bandwidth  limitations  in  teleoperation 


R.  Brockett 
10/12/96 

4th  International  Conference  on  Hybrid  Systems 
Ithaca  NY 

Title:  Learning  Boolean  Expressions  via  Principal  Components 

R.  Brockett 
11/14/96 

University  of  British  Columbia 
Vancouver,  BC,  Canada 

Title.  Modeling  Intelligent  Systems:  Languages,  Automata,  and  Differential  Equations 


R.  Brockett 
11/19/96 

Brown  University 
Providence,  RI 

Title:  Performance  Enhancement  Through  Optimal  Allocation  of  Effort 

R.  Brockett 
3/2/97 

Joint  Conference  on  Information  Systems 
Duke  University 
Durham,  NC 

Title:  Near  Optimal  Control  with  Communications  Constraints 

R.  Brockett 
3/23/97 
GAMM  97 

University  of  Regensburg 
Regensburg,  Germany 

Title:  Mathematical  Approaches  to  Learning  Applied  to  Control  Systems 


R.  Brockett 
4/8/97 
ASYNC  97 

Eindhoven  University  of  Technology 
Eindhoven,  the  Netherlands 

Title:  Methods  of  Analysis  for  Asynchronous  Systems 

R.  Brockett 
5/29/97 

Momingside  Center  of  Mathematics 
Chinese  Academy  of  Sciences 
Beijing,  China 

Title:  Lectures  of  Nonlinear  Control  Systems 


(h)  contributed  presentations 


(i)  transitions  of  ideas  to  industry  or  military 

On  November  12,  1996,  Stanford  graduate  student  M.  Costa  collaborated  with  researchers  at  the 
NASA  Ames  Intelligent  Mechanisms  Group  (IMG)  to  conduct  a  field  test  of  a  stylus  (fingernail) 
sensor  developed  at  Stanford.  The  test  was  conducted  on  the  Marskohod  rover  at  a  remote  site  in 
the  Arizona  desert.  The  stylus  was  attached  to  an  end-effector  on  the  rover  and  used  to  scratch 
the  surfaces  of  rocks.  Data  collected  by  the  stylus  were  sent  via  satellite  to  Mountain  View,  CA 
where  they  were  displayed  to  geologists  through  a  haptic  interface.  The  experiment  demonstrated 
the  potential  utility  of  haptic  feedback  as  an  adjunct  to  other  sensory  channels  (e.g.,  video)  for 
remote  field  geology. 

In  August  1997,  the  Stanford  Dextrous  Manipulation  Lab  became  a  subcontractor  to  Virtual 
Technologies  Inc.  of  Palo  Alto,  CA,  under  ONR  STTR  Topic  Number  N96T003:  “A  Grasp  and 
Arm  Force  Feedback  System.”  In  this  project,  Virtual  Technology’s  CyberGlove  is  equipped 
with  a  prototype  force  feedback  apparatus  and  used  to  control  a  two-fingered  manipulator  in  the 
Dextrous  Manipulation  Lab.  The  goal  of  the  joint  experiments  is  to  quantify  the  efficacy  of 
“ungrounded”  finger  force  feedback,  and  to  develop  teleoperation  methods  that  map  between  a 
gloved  human  hand  manipulating  virtual  objects  and  a  robot  hand  manipulating  real  objects. 

In  June  1997,  Stanford’s  Dextrous  Manipulation  Laboratory  entered  into  a  contract  with  Interval 
Research  corporation  of  Palo  Alto,  CA  to  investigate  the  tactile  and  kinesthetic  properties  of 
devices,  such  as  miniature  mechanisms,  for  which  the  dynamics  are  dominated  by  friction.  The 
project  involves  the  design  of  a  haptic  interface  that  will  be  used  to  identify  the  dynamic 
properties  of  devices,  including  a  model  of  the  friction,  and  to  render  the  models  of  such  devices 
to  human  subjects. 

In  August  1997,  the  Stanford  Dextrous  Manipulation  Lab  became  a  subcontractor  to  Immersion 
Inc.,  San  Jose  CA,  under  an  STTR  project  on  an  “Advanced  Reconfigurable  Haptic  Interface 
System.”  The  goal  of  this  project  is  to  develop  methods  for  efficiently  identifying  and 
representing  the  surface  properties  of  objects  in  a  remote  or  virtual  environment.  The  work  will 
include  the  development  of  exploratory  procedures,  and  the  development  of  data  representations 
suitable  for  describing  objects  with  texture  and  fine  features. 

The  Harvard  Robotics  Lab’s  image-based  surface  reconstruction  technology  has  lead  to  the 
development  of  an  endoscopic  probe  for  use  in  minimally  invasive  surgery.  This  is  in 
cooperation  with  the  Harvard  Center  for  Minimally  Invasive  Surgery  at  Deaconess  Hospital, 
headed  by  Dr.  Armour  Forse. 

Harvard  University  and  Immersion  Corporation  have  collaborated  on  a  Phase  I  STTR  (ONR 
contract  N00014-96-C-0325),  entitled  “Commercialization  of  Vibrotactile  Feedback  for 
Telemanipulation  and  Virtual  Environments.”  This  project  involves  collecting  vibration  data 


resulting  from  using  a  stylus  to  tap,  scratch,  and  puncture,  various  materials,  textures,  and 
membranes.  Empirical  models  were  fit  to  these  data  to  create  a  library  of  waveforms. 
Preliminary  results  of  this  integration  indicate  that  vibration  feedback  enhanced  operator 
performance  for  several  different  tasks.  An  STTR  Phase  n  proposal  to  realize  commercial 
prototype  systems  using  vibrotactile  feedback  has  been  submitted. 

(j)  training  data 

Stanford: 

Undergraduates:  0 

Graduate  students:  5(1  woman,  0  non-US  citizen,  3  minority) 

Post  Docs:  0 

Harvard: 

Undergraduate  Students:  8 

Graduate  Students:  4(1  Woman,  1  non-US  citizen) 

Post-docs:  2(1  non-US  citizen) 

(k)  awards  and  honors 

(l)  cost  and  descriptions  of  items  exceeding  $1000 

Stanford: 

IBM-compatible  computer  system  for  haptic  interface  programming:  $3500 
Inunersion  Inc.  Impulse  Engine  haptic  interfaces  and  associated  software:  $5000 


Harvard: 

ELMO  QN401  Microcamera:  $3264 

EPIX  4MIP  Image  Processing  and  Acquisition  Board:  $508 1 


Grounded  vs.  Ungrounded 
Haptic  Feedback 

Grounded  Ungrounded 


Forces  applied  with 
respect  to  a  stationary 
reference  frame 

Examples: 

PHANToM,  Impulse  Engine 

Absolute  force  reference 
Accurate  rendering  of 
stationary  objects 

Limited  Workspace 


F 


Forces  applied  locally 
with  respect  to  arm  or  hand 

Examples: 

Cybergrasp,  Rutgers  Master  II 

Large  Workspace 
Portable 


Cannot  Impede  arm  motion 


Question:  How  effectively  can  ungrounded 
devices  display  contact  with  stationary  objects? 


Figure  1 . 


Figure  2.  Example 
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Figure  3.  Typical  results  from  [Richard  and 
Cutkosky,  1997]  for  object  discrimination  with  three 
types  of  feedback:  A)  ungrounded  B)  grounded  c) 
grounded  with  fingertip  forces. 
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Figure  4 


